The present invention relates to graphical and auditory presentation of data, particularly for exploiting human pattern recognition.
The field of information visualization includes the use of graphs to convey information in a useful manner. Appropriate information displays can bring life to otherwise inert matrices or streams of numbers. Human perception and recognition of data trends can be facilitated through construction of comprehensible graphs. This trend perception is especially important when given large amounts of multi-variate data from which useful information must be quickly derived. Depending upon the environment and the importance of the decisions to be made, even the best information can become overwhelming. An example of one such environment is a fast-paced stock trading room where financial analysts are expected to quickly assess various online sources of information and make irrevocable decisions that can effect their very careers. Other situations arise in civil emergencies where an uninformed decision could jeopardize lives. The computer industry has generated a number of tools for coping with problems such as these.
Our brains allow us to do many things that defy even the most complex artificial systems. At the fore is our ability to organize a diverse range of information into relatively simple patterns and to monitor these patterns for outliers; cases that break ranks with the majority. The brain is the best xe2x80x9cpattern detectorxe2x80x9d in existence. The conventional data visualization tools have failed to take advantage of the various cognitive and perceptual powers of the brain in any structured manner.
A graph is a visual display that illustrates one or more relationships among numbers. The best graphs are those that permit a visual pattern, trend or comparison to be quickly and accurately comprehended by a human reader. A poorly designed or constructed graph can be difficult to decipher properly, and could result in contusion or erroneous conclusions. Based upon knowledge of the particular audience, coupled with an understanding of human perception and cognition, a certain craftsmanship can be brought to bear upon the task.
Application of empirical findings from research on human cognition and perception to creation of graphs is explored in Stephen M. Kosslyn""s book, xe2x80x9cElements of Graph Design,xe2x80x9d (W.H. Freeman and Co., 1994), which is incorporated herein by reference. The author evaluates many of the factors that should be considered when selecting a graph format to present specific content for a specific purpose, and derives a set of principles having a basis in the physiological psychology of human cognition and perception. Exploitation of these principles when formulating a graphical presentation can dramatically improve the useful information content without appreciable increase in visual complexity. The author derives a list of principles that provide a framework against which to calibrate the relative effectiveness of various approaches to data visualization. In general, the principles can be divided into three sections: one regarding the way in which we actively organize and interpret what we see; another regarding how meaning is derived from visual displays; and a third related to memory and processing limitations related to proper interpretation. These same principles also apply directly to patterns of sounds.
One approach for display of information is disclosed in U.S. Pat. No. 5,671,381 issued to Strasnick. A three-dimensional, virtual reality display space is created to contain objects that represent blocks of data as 3-D bar-charts. Attributes of the data are mapped to visible or audible characteristics, such as an icon having a specific size or color. The spatial relationship and connecting lines between icons in the landscape represent structural relationships that exist in the underlying data, with the ground plane representing a numerical value as a common surface plane. Artificial perspective (with object compression near the horizon) adds to the realism of the view. A user can arrange the objects according to a preferred lexical order, and then xe2x80x9ctourxe2x80x9d across the landscape to browse or search for particular data items or relationships.
There are many disadvantages of this method of visualization. For one thing, the data representations are purely artificial, much like a two-dimensional bar-chart with its necessarily limited information bandwidth. A static hierarchical tree paradigm dictates the arrangements between the icons, illustrated by cluttered linkage lines. In essence, the user can merely navigate through a sea of bar charts illustrating the size and age of files, or similarly benign parameters. There is no means for indicating any data changes, nor their magnitude, relevance, or direction. Given the lack of change information, it would be impossible to detect any patterns of changes among the data sets. Furthermore, sound is implemented solely as a xe2x80x9cwarning tonexe2x80x9d triggered when the user""s cursor touches a file icon having a predetermined attribute. There is no selection from a variety of meaningful sounds, or any concept of spatial orientation, intensity, or inherent recognition of the sound""s meaning, other than its mere occurrence.
Significantly, the system relies upon the virtual (i.e., artificial) reality context, a computationally complex, and visually distracting data display. Virtual reality (VR) attempts to replicate physical reality, where the better the VR system, the better the rendition. Visualizations of this type accentuate the details at the expense of data comprehension. As described in Kosslyn""s xe2x80x9cElements of Graph Designxe2x80x9d, human users can reliably process only a limited amount of visual information at one time, depending on a number of psychological factors including relevance and alertness. It takes additional effort for the mind to construct a 3-D perceptual organization of random icon meanings and orientations, especially when they are made even less intelligible by the artificial variations and distortions constantly introduced by xe2x80x9cnavigating.xe2x80x9d A VR display contaminated with irrelevant or overly detailed information may actually reduce the ability to properly perceive the data patterns of most interest.
A similar arrangement, specifically addressed to visualization of information useful to money managers, is disclosed in U.S. Pat. Nos. 5,675,746, and 5,774,878, both issued to Marshall. In these and related patents, so-called 3-D xe2x80x9cmetaphorsxe2x80x9d are used to represent data in a virtual reality setting, where characteristics of each metaphorical object are determined by the corresponding data. The shape, color and rotation of each object may change according to the data, or to highlight criteria selected by the user. The location of the object may represent the source of the information (e.g., a selected market information feed), or a collective similarity (e.g., industry groups). The user may then xe2x80x9cflyxe2x80x9d among the objects to observe their characteristics more closely. For example, objects xe2x80x9cfloatingxe2x80x9d above the perceptual ground plane could represent data for stocks trading xe2x80x9cabove average.xe2x80x9d The respective meaning of sounds, shapes and movement of 3-D objects are specified by the system configuration, although no particular arrangements are described.
Many of the disadvantages of this system relate to the limited visualization mechanisms employable. The arrangement of icons is according to a predetermined set of three-dimensional axes. The icons themselves have corresponding shape, size and color that are purely arbitrary and which lend very little to any inherent perception of their respective value. The portrayal of spinning, colored or pulsating icons merely represent data or data trends that have already been calculated. The totally abstract landscape does not lend itself to evaluating any recognizable objects, let alone interactions or patterns. There is nothing in the virtual reality arrangement to facilitate recognition of xe2x80x9coutliersxe2x80x9d, i.e., the few non-conforming data sets. Each of these limitations of the prior art fosters a requirement for a large amount of learning before the display begins to become useful for anything more than a data browsing tool.
In addition to the other disadvantages of virtual reality mentioned above, a user unfamiliar with the medium not only must learn to configure and navigate the data, but can easily become overwhelmed or disoriented by the lack of uniformity or structure. At most, the Marshall system is disadvantageously grounded on depth perception, and spatial intuition, which do not recognize the limits of human cognition, memory, or comprehension. Without these elements properly constrained, the virtual reality icons remain relatively indecipherable, and the information representations frustratingly inscrutable. Furthermore, the use of sound by Marshall has the disadvantage that it is used only for manually identified and selected/highlighted cases rather than showing patterns or any other information general to the data set. Even when sound is implemented, there is nothing to indicate how the sound can be used to provide multivariate texture to the audible display.
In an effort to bring a physical association to data visualization techniques, U.S. Pat. No. 5,321,800, issued to Lesser, describes a graphical method tied to medical data about a patient. A single display template is arranged as a representation of the human form, with binary information icons imposed upon fixed locations of the form. The present data values are illustrated by varying the color or intensity of a specific anatomical location of the display corresponding to the data item. Once a user (i.e., nurse or doctor) has thoroughly learned the location and meaning of the various elements, the theory is that the information about the present patient can be quickly deciphered. For example, arm color of blue may represent the corresponding location of an infection, and specific blocks of colors on the chest or abdomen may represent recent blood-testing results, vital signs, patient complaints (e.g., pain), or other diagnostic observations. Data excursions beyond predetermined thresholds can be indicated by blinking red icons at the plotted location.
Among the many disadvantages of the Lesser system is that, like Marshall above, the shapes and changes in the icons provide precious little if any information that capitalizes upon human perception, memory or comprehension. The proposed placement of rectangular blocks of data is completely arbitrary, other than the obvious anatomy-oriented locations dictated by the patient data itself. Even the examples described in the illustrative embodiment demonstrate a general disregard for human cognition. For example, everyone knows that an area of infection is xe2x80x9cred,xe2x80x9d because of inflammation, yet Lesser would indicate such a state with a blue icon. Similarly, the data icons are comprised of fixed size, shape and location, using combinations of three colors: red, green, and yellow, to represent xe2x80x9cdeviationsxe2x80x9d from a normal value. This severely restricts the Lesser approach to types of data that can be characterized as normal or abnormal conditions. It also limits the information to a level of abstraction that relies upon the system parameters and requires specific assumptions be made about the patient to predefine the binary thresholds of such normality. Similarly, Lesser makes no reference to sound at all, other than xe2x80x9cbowel soundsxe2x80x9d the status of which may be reflected in a correspondingly colored icon. Ignoring this multimedia dimension further limits the amount of status information that can be derived.
Furthermore, a xe2x80x9cLessergramxe2x80x9d as taught by Lesser indicates the underlying data for one case at a time, not large numbers of cases. The design of Lesser fails to capitalize on any pattern relevance that would be cognizable across more than one data set, let alone hundreds. There is no xe2x80x9canalogical mappingxe2x80x9d of data in which completely different objects represent values of different measurements. Lesser uses the same rectangular shape over and over again for dozens of different types of data, dramatically aggravating an already steep learning curve.
Although Lesser does suggest superficially that a LesserGram could be applied to xe2x80x9cbusiness dataxe2x80x9d or other forms of information, there is no suggestion whatever that such a display could apply to anything more than purely physical, process-oriented data. One must presume, given the one-to-one relationship of data and display elements, that a LesserGram would require a fixed physical shape of some sort, determined by the process itself. Colored pieces would then portray some binary data status of the process being monitored. Again, this has nothing at all to do with defining icons or data representations to exploit the advantages or limitations of human memory, perception, or comprehension. At the very least, anyone familiar with one type of LesserGram for human medical information would have no idea how to apply the same technology to a different type of data, say, air traffic control, or weather.
One other example of using matrices of rectangles with different shapes and colors is in use by SmartMoney.com of New York, NY, to illustrate stock market data. According to their information, Market Today, a user can see xe2x80x9cThe market at a glance.xe2x80x9d Information about six hundred publicly traded stocks is arranged into industry segments. Individual market capitalization of each company dictates the relative size of a rectangle representing a company, and blocks within an industry segment that have similar histories of pricing movements are arranged next to each other. During a trading- session a user can request an update of the display when current information is desired. The color of each block varies in twelve steps from bright red, to black, to bright green (or another selectable spectrum, blue/yellow), indicating the present change in price with respect to a selected baseline (e.g., yesterday""s close).
Significantly, the market map demonstrates many of the disadvantages of visual information overload, without actually highlighting the news. Rectangles dictated by the nearly two thousand data points are squeezed into seemingly arbitrary locations on a display screen and then six hundred data points of a single metric (i.e., daily change) are simultaneously varied at the will of the market. Given the resulting confusion of data, the average user cannot be reasonably expected to quickly perceive changes, let alone patterns, unless several large, adjacent stocks happen to change simultaneously. The equivalent percentage change in a dozen xe2x80x9csmallxe2x80x9d stocks would not even be visible, albeit noteworthy, unless the changes all happened within the same arbitrary industry segment (i.e., adjacent rectangles), and only if the user happened to be closely monitoring that area of the map. This defeats the whole purpose of a usable monitoring tool.
The market map display is disadvantageously static. There is no display motion or audible representation to indicate any sort of change, nor whether the change is relatively good or bad, only that the underlying data is presently up or down from the start. In fact, there is no indication at all that anything in the display itself has changedxe2x80x94the user must attempt to remember what was previously displayed and mentally compare all 600 stocks (or at least some of them) when the display is next refreshed. Transitions in the market map rectangles from red to black to green are shown using more than ten variations in color, forcing a user to perceive, recognize, recall, and understand the subtle differences, a daunting (if not impossible) task when under immense pressure.
Paradoxically, since the data icons never change size or shape, the method limits the amount of visual information that could be usefully displayed about any particular datum. Aside from the fact that SmartMoney provides transitional information about only a single data metric (i.e., percentage change since last close), it has several other serious drawbacks with respect to data visualization. The market map is constructed using the fixed hierarchy of industry segment, market caps, and percentage changes. A user is helpless when it comes to organizing the display according to other metrics that may be preferred by the user. Even if the viewer could select stocks or specific arrangements or collections for monitoring, the only thing that ever changes is the colors. The map of the market does not take any advantage of the abilities of the human mind to perceive, memorize, and comprehend limited amounts of salient information, or to detect changes of display motion.
The present invention provides a new method whereby combinations of xe2x80x9cecologically validxe2x80x9d (i.e., occurring in everyday environments) images and sounds are used for visualization of large amounts of information. Icons and sounds are selected and combined to accentuate natural recognition of pattern information. A visual field of similar icons represents a corresponding field of data sets, where the appearance of each icon illustrates the relative values of the underlying data. Within each icon, elements change in size, number, color, and motion to illustrate the respective directions and magnitude of changes of data within the corresponding data set. Stereophonic sound occurrence, timing, spatial location, type, and volume signal the user about general data situation, as well as the type and importance of individual data changes, stimulating the user""s attention during events of significance.
According to an illustrative embodiment of the system, an icon suggesting a recognizable life form is selected as the basis for visualization of a multi-variate data set. Life forms, such as flora and fauna, are known to grow, move and change colors according to their inherent physical determinants. For example, a healthy flower such as a daisy has a green stem, and gets larger and taller, and has more petals, as naturally determined by the flower""s environment. A shorter, brown flower that has lost most of its petals would be instantly recognized as produced by a less satisfactory condition. Furthermore, a flower can wave in proportion to the breeze, and the fertility of the area around the base of a flower may convey additional information regarding the local trends.
Similarly, an auditory display of the system selects sounds that are easily recognizable as positive or negative (i.e., good/bad). The sounds are then produced in loudness, location, and timing patterns according to the underlying data. For example, loud and frequent bird-chirping sounds on the right side would indicate a generally positive environment, whereas a rumbling of thunder to the left would give a more negative impression. Musical sounds of various instruments and tones could also be used instead of, or in addition to animal or weather sounds.
By using an ecologically oriented graphical icon, a wide number of different parameters can be simultaneously depicted. Motion and sound are used to highlight important changes, or to lend assurance that a status quo prevails.
In an illustrative embodiment, an icon of a daisy is configured to represent financial data such as relative change in stock price from opening, absolute stock price, standard deviation in price, direction of change in price, volume of trades, and short-term trends. These parameters are mapped to the number and size of petals, the frequency of swaying motion, the color and length of the stem, and the color of the grass near the stem. The magnitude of change is represented by the waving motion, where bigger changes make the waving faster. Similarly, when a sound occurs, the appropriate flower xe2x80x9clights upxe2x80x9d, and pulses for a predetermined period with green for a higher trade, red for a lower trade. The stock symbol can also appear next to such blinking flowers for a predetermined time. Hundreds of such xe2x80x9cdaisiesxe2x80x9d can then be arranged on a display, and sorted (e.g., top to bottom) depending upon a user-selectable choice of display parameter. These flowers can be further sorted into quadrants of different types of flowers, not just daisies, with the type of flower representing a categorical variable, such as stocks from a particular industry. Positive and negative sounds are then superimposed on the display according to whether stocks are being traded for prices above or below their respective averages.
Unlike the prior art, the present method includes ecologically valid icons with which a human is likely to be familiar. This method includes a visual uniformity, salient data icons, low clutter yet higher dimensionality, and sound overlays. In addition, there are alerts to changes (both visual and auditory), a palette of tones, and selectable data for either a binary condition (red/green) or a scaled measure of variations. Little superfluous information is displayed, unlike VR approaches, and since less detail is needed, more information becomes recognizable. There is no xe2x80x9cchange for change sakexe2x80x9d since nothing moves unless the corresponding data has changed. Lower attention is necessary since you""re not xe2x80x9cflyingxe2x80x9d anything, but rather planted firmly on the grass background with the flowers. This method exploits human mental powers of perception, memory and cognition. The data icons can also be simply sorted (e.g., top to bottom) by user selectable parameters to give inherent order to the depiction without the inflexibility of predefined dimensionality.
Outliers are readily apparent from the behavior of the icons, especially where large changes are indicated by proportional changes in sound, color, and motion. Like picking out the image of a rabbit in a wheat field, the human mind quickly tunes out the irrelevant visual data. The use of ecologically valid visual icons reduces the learning time and operator confusion and capitalizes on the cognitive abilities of the human viewer.
Similarly, the use of natural sounds facilitates recognition of positive or negative circumstances or present changes. The location, volume, selection, duration and other attributes of the sounds are easily mapped by the brain to represent the corresponding information regarding the magnitude and type of changes simultaneously occurring in the visual data. The presence of the sounds allows the user to look away from the computer screen, while being able to listen to sounds that indicate a change in the data that may require visual attention.
Unlike Lesser, or other process-oriented visualization methods, the ecologically valid visual and audible icons can be easily adapted to represent numerous types of data for which patterns and changes must be quickly and efficiently recognized. The analysis is divorced from the one-to-one correspondence for which intuition or training are imperative, as would be required when using a LesserGram. The same operator, using the same icons, can apply the tool to many different types of data, and detect the necessary information quickly, be it weather phenomena, stock markets, or other data intensive analyses, without necessarily understanding the underlying processes that create the data, and unencumbered by the analytical prejudices spawned by such familiarity.
Displays created according to the inventive method are intentionally limited with respect to the amount of information that can be directly determined from inspection, in an effort to focus upon the most relevant factors, their changes, and the direction and magnitude of such changes. Because of the uniformity of the many icons, they can quickly be compared with each other for changes, while still observing general trends, if necessary. Direction of change is binarized, with the magnitude displayed only for selected changes (petals, stem, speed, sound volume), thus limiting the amount of refined discrimination required by the user. Changes are indicated not only with visual movement calculated to stimulate the visual perception per se, but also with the simultaneous sound occurrence. Movement speed, sound volume and location, and sound selection draw the user""s attention to the other instantaneous changes that have occurred (petals, stem length). Thus, the occurrences of multidimensional changes are quickly recognized, along with the outliers and interactions with other data sets.